Papers with dual encoders
Sparse, Dense, and Attentional Representations for Text Retrieval (2021.tacl-1)
Copied to clipboard
| Challenge: | Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. |
| Approach: | They propose a dual-encoder-based neural model that combines the efficiency of dual encoders with expressiveness of more costly attentional architectures. |
| Outcome: | The proposed model outperforms strong alternatives in large-scale retrieval. |
How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)
Copied to clipboard
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran
| Challenge: | generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones . |
| Approach: | They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora . |
| Outcome: | The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size . |
What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary (2023.acl-long)
Copied to clipboard
| Challenge: | Dense retrieval models based on text representations have proven very effective, but when applied off-the-shelf they often experience a severe drop in performance. |
| Approach: | They propose to interpret the vector representations produced by dual encoders by projecting them into the model’s vocabulary space. |
| Outcome: | The proposed model significantly improves on the BEIR benchmark and in zero-shot settings. |
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | ConveRT is a pretraining framework for conversational AI that is computationally heavy, slow, and expensive to train. |
| Approach: | They propose a pretraining framework for conversational tasks that is efficient, lightweight, and inexpensive. |
| Outcome: | The proposed model achieves state-of-the-art performance across widely established responses . it trains substantially faster than existing state- of-the art models . |
Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward (2020.acl-main)
Copied to clipboard
| Challenge: | Abstractive summarization models for document encoders suffer from fabricated content and are often near-extractive. |
| Approach: | They propose a framework for abstractive summarization with Graph-Augmentation and semantic-driven RewarD that uses a sequential document encoder and a graph-structured encoder to maintain the global context and local characteristics of entities. |
| Outcome: | The proposed framework produces higher ROUGE scores than a variant without knowledge graph on New York Times and CNN/Daily Mail datasets. |
Crossing Variational Autoencoders for Answer Retrieval (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods learned semantic representations with dual encoders or dual variational auto-encoders failed to capture the aligned semantics between question and answer. |
| Approach: | They propose to use two variational auto-encoders to generate questions with aligned answers and generating answers with align questions. |
| Outcome: | The proposed method outperforms the state-of-the-art answer retrieval method on SQuAD. |
Exploring Dual Encoder Architectures for Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Dual encoders have been used for question-answering and information retrieval tasks with good results. |
| Approach: | They propose to use two different versions of dual encoders for QA retrieval tasks . they propose to share parameters in projection layers between two encoder towers . |
| Outcome: | The proposed architectures outperform SDE and ADE on QA retrieval tasks. |
Large Dual Encoders Are Generalizable Retrievers (2022.emnlp-main)
Copied to clipboard
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, Yinfei Yang
| Challenge: | Experimental results show that dual encoders outperform sparse and dense retrievers on the BEIR dataset significantly. |
| Approach: | They challenge belief that bottleneck layer is too limited for out-of-domain generalization . they scale up the model while keeping bottleneck as a single dot-product with a fixed size . |
| Outcome: | The proposed model outperforms sparse and dense retrievers on the BEIR dataset significantly. |
SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives (2023.findings-acl)
Copied to clipboard
| Challenge: | Dual encoders have been used for retrieval tasks and representation learning with good results. |
| Approach: | They propose an improved contrastive learning objective by adding queries or documents from the same encoder towers to the negatives. |
| Outcome: | The proposed model improves retrieval quality for both symmetric and asymmetric dual encoders by adding queries or documents from the same encoder towers to the negatives. |
Coarse-to-Fine Dual Encoders are Better Frame Identification Learners (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent efforts to model frame definitions lack sufficient representation learning of definitions or lack efficient frame modeling. |
| Approach: | They propose a frame-target-encoder architecture that uses coarse-to-fine learning to model alignment between frames and targets. |
| Outcome: | The proposed framework outperforms existing models by 0.93 overall scores and 1.53 R@1 without lf. |